Calculating the Nml Distribution for Tree-structured Bayesian Networks

We are interested in model class selection. We want to compute a criterion which, given two competing model classes, chooses the better one. When learning Bayesian network structures from sample data, an important issue is how to evaluate the goodness of alternative network structures. Perhaps the most commonly used model (class) selection criterion is the marginal likelihood, which is obtained by integrating over a prior distribution for the model parameters. However, the problem of determining a reasonable prior for the parameters is a highly controversial issue, and no completely satisfying Bayesian solution has yet been presented in the non-informative setting [1]. The normalized maximum likelihood (NML), based on Rissanen’s information-theoretic Minimum Description Length MDL methodology [2, 3, 4], offers an alternative, theoretically solid criterion that is objective and non-informative, while no parameter prior is required. The NML distribution has several desirable properties. Firstly, it automatically protects against overfitting in the model class selection process. Secondly, there is no need to assume that there exists some underlying “true” model, while most other statistical methods do: in NML the model class is only used as a technical device to describe the data, not as a hypothesis. Consequently, the model classes amongst which to choose are allowed to be of utterly different types; any collection of model classes may be considered as long as the corresponding NML distributions can be computed. For this reason we find it important to push the boundaries of NML computability and develop algorithms that extend to more and more complex model families. It has been previously shown that for discrete data, this criterion can be computed in linear time for Bayesian networks with no arcs [5], and in quadratic time for the so called Naive Bayes network structure [6]. Here we extend the previous results to tree-structured networks. We show how to compute the NML criterion in polynomial time for Bayesian Forests, i.e. the family of Bayesian models where in the corresponding network structure no node has more than one parent . The order of the polynomial depends on the number of values of the variables, but neither on the number of variables itself, nor on the sample size.